Evaluating Progress in Probabilistic Programming through Topic Models

Topic models have proven versatile over the past decade, particularly as partial embeddings within more intricate models. These models present challenges that are analytic, computational and engineering in nature. Advances in probabilistic programming have the potential to circumvent a number of these issues, but researchers need a way to coarsely evaluate these frameworks. We identify three axes of a successful framework and argue that computational efficiency of straight LDA provides one such lens. We provide and release a modular open-source testbed to systematically capture one aspect of current probabilistic programming and discuss initial results on both heavily-controlled and “real” data.